Apparatus and method for tracking hand-held surgical tools

ABSTRACT

Determining location and orientation of a hand-held surgical tool is provided. One embodiment captures image data that includes images of a first detectable target on the hand-held surgical tool and a second detectable target on a patient. Location and orientation or the first detectable target in a 3D space is determined based on the image data. Current location and orientation of the hand-held surgical tool in the 3D space determined based on the determined location and orientation of the first detectable target and based on retrieved model data representing the hand-held surgical tool. Location of the second detectable target in the 3D space is determined based on the image data. Then, a location of the patient&#39;s tissue in the 3D space relative to the location and orientation of the hand-held surgical tool is determined based on the determined location of the patient&#39;s second detectable target.

BACKGROUND OF THE INVENTION

During medical interventions. There is a great need to be exact in thepositioning of hand-held surgical tools relative to tissues that thepractitioner is going to be interacting with. Typically, thepractitioner concurrently views both the working end of the hand-heldsurgical tool and the viewable portion of tissue that the practitioneris interacting with. However, the practitioner may not be able tovisually see some tissue of interest and/or portion or the working endof the hand-held surgical tool that are not exposed to view.

Various electronic devices and systems have been developed to providevisualization aids to the practitioner while they are using a hand-heldsurgical tool to interact with tissue or a patient. Some systems providevisualization or the position of the working end of the hand-heldsurgical tool by presenting images on a display that is viewable by thepractitioner. Sensing location and orientation of the working end of thehand-held surgical tool is typically done using ultrasound devices,computed tomography (CT) scanning devices, x-ray devices, fluoroscopyscanning devices, and/or magnetic resonance imaging (MRI) system. Eachof the modalities comes with its own limitations. For example, X-raymachines, MRI machines and CT scanning machines are expensive to acquireand to use. X-ray images present graphical information on a limitedtwo-dimensional-plane. MRI is unsatisfactorily slow and supplies lowresolution images of bone structures and are not very practical in anoperating theater or during procedures in outpatient setting.

Further, X-ray imaging, fluoroscopy and CT scanning use ionizingradiations (X-rays) that may be harmful to the patient and thepractitioner who is performing a surgical procedure in the surgicaltheater. The longer the surgical procedure takes, the more potentiallyharmful radiation both the patient and the practitioner will besubjected to. For example, a patient that is getting an epiduralinjection in the spine will have to undergo a video fluoroscopy for anextended period of time. The practitioner may have to wear cumbersomelead shielding. The patient will also have to use a suitable leadshield. Further, the fluoroscope is usually exceptionally large and canleave only a small space available for the practitioner to work.Intra-operative CT such as cone beam will have similar downsides.

Other less potentially harmful electronic scanning systems and devicesare available for acquiring patient information. For example, ultrasounddevices project sound waves into the patient and detect returning soundwave echoes that are used to generate an image, referred to as asonogram image. Ultrasound devices used in ultrasonographic systemsproduce sound waves at a frequency above the audible range of humanhearing, which is approximately 20 kHz. Sound waves between 2 and 18 MHzare often used for ultrasound medical diagnostic applications. Atpresent, there are no known long-term side effects from interrogatingthe human body with ultrasound waves.

However, an ultrasound scan can cover only a relatively small part ofthe patient's body with each scan. Further, the sonogram is a relativelynarrow image, covering a relatively small cross-section of only a fewinches. And objects found in the sonogram image may often be blurry. Forexample, five hundred to one thousand sonogram images must be capturedto acquire enough image data for analysis of a full human spine.Accordingly, legacy ultrasound scanners are inadequate for acquiringimage Information for the patient's body when a large area of the humansubject must be examined, such as the patient's spine, because thesonogram images are too small, and many sonogram images cannot be easilyanalyzed to arrive at any meaningful information about the condition ofthe examined patient.

There is also a problem with the differentiation of the tissue that isviewed in a sonogram image. That is, it is exceedingly difficult for thepractitioner to tell which tissue they are viewing in a sonogram image.Typically, when a surgical procedure is performed on area near thespine, for example, the ultrasound system will only provide twodimensional (2D) ultrasound images. Even if real-time acquired sonogramimages provide visual information that indicates the current locationand orientation of a hand-held surgical tool during the surgicalprocedure, the practitioner still will have difficulty identifying sometypes of tissue in a presented sonogram image. Further, the practitionerwill have difficulty identifying the location and/or orientation of theworking end of the hand-held surgical tool with respect to the tissue(because the tissue is difficult to identify in a 2D sonogram image).

The inventors have created an ultrasonagraphic system that is operableto acquire sonogram information from a series of ultrasonic scans of apatient, as disclosed in U.S. Pat. Nos. 9,675,321 and 9,713,508, whichare both incorporated by reference herein in their entirety. Inpractice, a series of time indexed ultrasound scans are taken over aportion of interest on the patient which has their underlying bonestructure or other ultrasound discernable organ that is underexamination. Location and orientation of the ultrasound scanner, andlocation of the scanned portion of the patient, are preciselyidentifiable in the time indexed sonogram images that are acquired fromthe sonogram scanner of the patient during the scanning process. Thedata from the series of acquired time indexed sonogram scans aresynthesized into a single data file that is used to generate athree-dimensional (3D) Image and/or 3D model of the underlying bonestructure or organ of the examined patient, referred herein as a tissuemodel. However, this system is unsuitable for an actual surgicalprocedure since the 3D model of the tissue of interest (bone and/ororgans of the patient) because the numerous individual sonogram scansused to generate a 3D tissue model must be acquired before initiation ofthe surgical procedure. The ultrasonagraphic system disclosed in U.S.Pat. Nos. 9,675,321 and 9,713,508 was not designed to sense location andorientation or a hand-held surgical tool during a surgical procedure.

Accordingly, there is a need in the arts to more effectively acquireimage data that is presentable to a practitioner in real-time indictingcurrent location and orientation of a hand-held surgical tool relativeto the tissue of a human subject during a surgical procedure.

SUMMARY OF THE INVENTION

Embodiments of the hand-held surgical tool tracking system provide asystem and method for tracking location and orientation of a hand-heldsurgical tool being manipulated by a practitioner to interact withtissue of a patient. Example embodiments determine location andorientation of a hand-held surgical tool by capturing image data thatincludes images of at least a first detectable target on the hand-heldsurgical tool and a second detectable target on the patient. Locationand orientation of the first detectable target in a 3D space isdetermined based on the image data. Current location and orientation ofthe hand-held surgical tool in the 3D space determined based on thedetermined location and orientation of the first detectable target andbased on retrieved model data representing the hand-held surgical tool.Location of the second detectable target in the 3D space is determinedbased on the image data. Then, a location of the patient's tissue in the3D space relative to the location and orientation of the hand-heldsurgical tool is determined based on the determined location of thepatient's second detectable target. In some embodiments, relativelocation of the hand-held surgical tool relative to a patient's tissueis determined. Then a composite image showing the hand-held surgicaltool and an image of the tissue (based on a predefined model of thetissue) is presented on a 2D and/or 3D display.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative toeach other. Like reference numerals designate corresponding partsthroughout the several views.

FIGS. 1A-1E are figures of various types of hand-held surgical toolswith one or more detectable targets at known locations on the surface ofthe hand-held surgical tool.

FIG. 2 is a schematic view of a hand-held surgical tool tracking systemfor assisting a practitioner performing a procedure on a human subject.

FIG. 3 is a conceptual illustration of a presented composite imageshowing the relative location of a spine of a patient and a hand-heldsurgical tool.

FIG. 4 is a conceptual diagram of an embodiment of the hand-heldsurgical tool tracking system and a robotic operation systemcooperatively working together to generate a composite image thatincludes robotic tools and hand-held surgical tools.

DETAILED DESCRIPTION

FIGS. 1A-1E are figures of various, non-limiting example types ofhand-held surgical tools with one or more detectable targets 102 a, 102b at known locations on the surface of the hand-held surgical tool 104a-e. In a preferred embodiment, the detectable targets 102 a, 102 b areoptically detectable, though the detectable targets 102 a, 102 b may bedetectable using non-optical systems. FIG. 2 is a schematic view of ahand-held surgical tool tracking system 100 for assisting a practitionerperforming a surgical procedure on a human subject 200 (interchangeablyreferred to herein as a patient 200).

During the surgical procedure, embodiments of the hand-held surgicaltool tracking system 100 determine location and orientation inthree-dimensional (3D) space of the hand-held surgical tool 104, and inparticular, the working end (tool end) of the hand-held surgical tool104. Accordingly, the location and orientation in the 3D space of thehand-held surgical tool 104 that is being manipulated by thepractitioner during a surgical procedure is determinable in real-time ornear real-time. Based on a predetermined model of the tissue of interestthat is being operated on by the practitioner, a real-time or nearreal-time rendering and presentation of an image showing the interactionof the working end 106 of the hand-held surgical tool 104 with atwo-dimensional (2D) or three dimensional (3D) model of the tissue ofthe patient 200 is presented on a display. Accordingly, the practitionermay intuitively understand the actual location of the working end 106 ofthe hand-held surgical tool 104 with respect to the tissue that is beingoperated on.

The presented image of the tissue and the hand-held surgical tool 104may be presented as a 2D image and/or a 3D image. Is some embodiments,the practitioner (or an assistant) wear virtual reality glasses to viewthe 3D image of the working end 106 of the hand-held surgical tool 104interacting with the tissue of interest. Multiple displays mayconcurrently present the 2D image and/or the 3D image so that otherparties may view the ongoing surgical procedure. Further, the timesequenced 2D images and/or a 3D images may be saved for later review.

The disclosed systems and methods for or assisting a practitionerperforming a procedure on a human subject 200 (FIG. 2 ) using thehand-held surgical tool tracking system 100 will become betterunderstood through review of the following detailed description inconjunction with the figures. The detailed description and figuresprovide examples of the various inventions described herein. Thoseskilled in the art will understand that the disclosed examples may bevaried, modified, and altered without departing from the scope of theinventions described herein. Many variations are contemplated fordifferent applications and design considerations, however, for the sakeof brevity, each and every contemplated variation is not individuallydescribed in the following detailed description.

Throughout the following detailed description, a variety of examples thehand-held surgical tool tracking system 100 are provided. Relatedfeatures in the examples may be identical, similar, or dissimilar indifferent examples. For the sake of brevity, related features will notbe redundantly explained in each example. Instead, the use of relatedfeature names will cue the reader that the feature with a relatedfeature name may be similar to the related feature in an exampleexplained previously. Features specific to a given example will bedescribed in that particular example. The reader should understand thata given feature need not be the same or similar to the specificportrayal of a related feature in any given figure or example.

The following definitions apply herein, unless otherwise indicated.

“Substantially” means to be more-or-less conforming to the particulardimension, range, shape, concept, or other aspect modified by the term,such that a feature or component need not conform exactly. For example,a “substantially cylindrical” object means that the object resembles acylinder, but may have one or more deviations from a true cylinder.

“Comprising,” “including,” and “having” (and conjugations thereof) areused interchangeably to mean including but not necessarily limited to,and are open-ended terms not intended to exclude additional elements ormethod steps not expressly recited.

Terms such as “first”, “second”, and “third” are used to distinguish oridentify various members of a group, or the like, and are not intendedto denote a serial, chronological, or numerical limitation.

“Coupled” means connected, either permanently or releasably, whetherdirectly or indirectly through intervening components. “Secured to”means directly connected without intervening components.

“Communicatively coupled” means that an electronic device exchangesinformation with another electronic device, either wirelessly or with awire based connector, whether directly or indirectly through acommunication network. “Controllably coupled” means that an electronicdevice controls operation of another electronic device.

“Real-time” means that real-time is instant, whereas “near real-time” isdelayed by some amount of time (whether that's by a few milliseconds oreven less.

In the various embodiments, like reference numerals represent likecomponents. For example, the reference numeral 104 may be used togenerically represent any hand-held surgical tool, and reference numeral104 a may be used to identify a specific hand-held surgical tool, suchas the example scalpel 104 a (FIG. 1A).

Optical targets 102 (interchangeably referred to herein as detectabletargets 102) may be any conventional, specially developed, or laterdeveloped optical target that is discernable by an optical trackingsystem of the hand-held surgical tool tracking system 100. In someexamples, the optical targets 102 may extend in three dimensions about athree coordinate axis and include distinct optical target portionsrepresenting each axis. In other examples, the optical target 102 mayextend in three dimensions about six axes and include distinct opticaltargets representing each of the six axes. Optical targets 102 arediscernably to the human eye and are discernable in a captured image(photographic image or video image).

The detectable targets 102 may be active, such as by emitting (orreflecting) electromagnetic signals or other energy signals from thedetectable target 102. Or the detectable target 102 may be passive, suchas a retro-reflective marker that reflects radiation energy emitted bysome interaction device. Such active or passive targets 102 aregenerically described herein as detectable targets 102 for brevity,though such detectable targets 102 may not be optically detectable by animage capture device. Any suitable detectable target 102 that is nowknown or later developed is intended to be within the scope of thisdisclosure and to be protected by the accompanying claims.

In the various embodiments, each of the detectable targets 102 areuniquely identifiable. In some embodiments, a visible characteristic ofan optical target (or at least one optical target) can be compared toknown identification characteristics of the optical target so aparticular optical target 102 is identifiable. For example, the shape ofthe optical targets 102 may be different. If the optical targets 102 aresmall marks, such as dots or the like, the number of marks for each ofthe optical targets 102 may identify that particular optical target 102.A unique alpha-numeric identifier may be located proximate to or on theoptical targets 102. Active detectable targets 102 may emit differentsignals with identifier information. In embodiments that employ only asingle optical target 102, that single optical target 102 is inherentlyidentifiable.

As conceptually illustrated in FIG. 2 , the non-limiting exampleembodiment of the hand-held surgical tool tracking system 100 comprisesa processor system 202, at least one hand-held surgical tool 104, and anoptional remote rendering and display system 204. The processor system202 comprises at least one image capture device 206, a target trackerunit 208, an image registration module 210, an image processingalgorithm module 212, a 3D/2D visualization module 214, a surgical tooldatabase 216, a tissue model database 218, and a user input device 220.Some embodiments include an optional clock 222. Some embodiments mayinclude an optional display 224. Any suitable processor system now knownor later developed is intended to be included within the scope of thisdisclosure and to be protected by the accompanying claims. In someembodiments, the processor system 202 may be communicatively coupled tothe remote rendering and display system 204 via a wire-based or wirelessconnection 226. Other embodiments of the hand-held surgical tooltracking system 100 may include some, or may omit some, of theabove-described components. Further, additional components not describedherein may be included in alternative embodiments.

In an example embodiments, the discernable targets 102 are opticaltargets 102. Here, the optical targets are discernable to a human viewerand are detectable in a photographic image. The image capture device 206is still or video camera device, and the acquired image data are stillimage photograph data or photographic video data.

FIGS. 1A-1E represent a non-limiting selection of example hand-heldsurgical tools 104 a-104 e. Any particular hand-held surgical tool mayinclude one or more detectable targets 102. Each type of hand-heldsurgical tool 104 is uniquely identifiable. An identifier of thehand-held surgical tool 104 may be located on each hand-held surgicaltool 104. The identifier may be an alpha-numeric identifier, a bar or QRcode, or the like that can be identified to the hand-held surgical tooltracking system 100.

For example, a captured image or a scan of the hand-held surgical tool104 may be analyzed by the hand-held surgical tool tracking system 100to determine the identifier of the hand-held surgical tool 104 using anysuitable alphanumeric text character recognition algorithm.Alternatively, or additionally, a radio frequency identifier tag (RFID)may be placed on the hand-held surgical tool 104 and/or its packagingthat can be read by an RFID reader provisioned within the hand-heldsurgical tool tracking system 100 or that is communicatively coupled tothe hand-held surgical tool tracking system 100. In some embodiments, anobject recognition algorithm may be used to identify the surgical tool104 based on captured image data. Alternatively, or additionally, thepractitioner or another party may manually specify the identifier of thehand-held surgical tool 104 to the hand-held surgical tool trackingsystem 100 (such as if the specification is entered using a keyboard, ifentered using a touch sensitive screen, or is spoken to an audibledetection and recognition system). Once the particular hand-heldsurgical tool 104 has been identified to the hand-held surgical tooltracking system 100, then hand-held surgical tool model datacorresponding to the identified surgical tool 104 (a model of thehand-held surgical tool model) can be retrieved so that the processorsystem 202 can determine the precise location and orientation of theidentified hand-held surgical tool 104, and in particular the workingend 106 located at the distal end of the hand-held surgical tool 104,during the surgical procedure.

Returning to FIG. 1A, the particular hand-held surgical tool 104 isappreciated to be a scalpel 104 a. The scalpel 104 a has a working end106 a (interchangeably referred to herein as a tool 106 a or blade 102a) that is understood by one skilled in the art to be the cutting edgeor blade 1061 located at the distal end of the scalpel 104 a. Prior touse of the scalpel 104 a during the surgical procedure, one or moreidentifiable optical targets 102 are placed at known locations (or arefabricated at the known locations) on the surface of the scalpel 104 ain the various preferred embodiments. Preferably, each of the opticaltargets 102 will be uniquely identifiable.

A variety of different types of detectable targets 102 may be used inthe various embodiments. A non-limiting first type is the exampleoptical target 102 a that has a plurality of distinct optical targetportions. When the optical target 102 a is viewed in a captured image(or more precisely, when image data corresponding to the optical target102 a is analyzed by the processor system 202), the location andorientation of that optical target 102 a can be precisely determinedbased on image characteristics of the image target portions (i.e.,relative location to each other). Since the orientation and location ofthe identified optical target 102 a is known with respect to allportions of the scalpel 104 a, and in particular to the location of theblade 106 a, the precise location and orientation of the blade 106 a inthe 3D space can be determined using a stored model corresponding to thescalpel 104 a.

In this non limiting example conceptually illustrated in FIG. 1A, amodel of the scalpel 104 a with data corresponding to the exampleoptical target 102 has been predetermined and stored. The location ofeach portion of the scalpel 104 a relative to the optical target 102 a,and in particular the precise location and orientation of the blade 106a to the optical target 102 a, is represented in the model data of thescalpel 104 a. When the acquired image data of the scalpel 104 a iscorrelated with the scalpel model data, embodiments of the hand-heldsurgical tool tracking system 100 determines the precise location andorientation of the scalpel 104 a, and in particular the precise locationand orientation of the blade 106 a, in 3D space.

Alternatively, or additionally, a plurality of optical targets 102 b maybe located on the surface of the hand-held surgical tool 104. Suchoptical targets 102 b may be simple marks, such as one or more smallcolored dots, squares, triangles, stars, or the like, that can beidentified in the image data acquired by the image capture device 206.Differing colors of and/or patterns on the mark may be used. Anysuitable shape, color and/or pattern may be used in the variousembodiments, particularly for mark identification purposes. The preciselocation and identity of each of the identifiable optical targets 102 bon the surface of the hand-held surgical tool 104 is known using astored model corresponding to the scalpel 104 a. With respect to thescalpel 104 a, two optical targets 102 b and 102 b′ are illustrated(wherein a characteristic of the optical target 102 b Is different fromthe optical target 102 b′ for identification purposes). Once at leasttwo optical targets 102 b are identified in the captured image data,then the precise location and orientation of the scalpel 104 a in 3Dspace can be computed (determined) by the processor system 202 based onthe identified relative location of the at least to optical targets 102b with respect to each other. Additional optical markers 102 b may beused to improve accuracy of the determined location and orientation ofthe scalpel 104 a.

In this non limiting example conceptually illustrated in FIG. 1A, amodel of the scalpel 104 a with the example optical targets 102 b and102 b′ has been predetermined and stored. The location of each portionof the scalpel 104 a relative to the optical targets 102 b and 102 b′,and in particular the precise location and orientation of the blade 106a to the optical target 102 b and/or 102 b′, is represented in the modeldata of the scalpel 104 a. When the acquired image data of the scalpel104 a Is correlated with the scalpel model data, embodiments of thehand-held surgical tool tracking system 100 determines the preciselocation and orientation of the scalpel 104 a, and in particular theprecise location and orientation of the blade 106 a, in 3D space. Oneskilled in the art appreciates that the optical targets 102 b and 102 b′are preferably uniquely identifiable so that the location andorientation of working end 106 of the hand-held surgical tool 104 isdeterminable. (If the optical targets 102 b and 102 b′ are notidentifiable from each other, it is possible that an incorrectorientation and location of the surgical tool 104 could be determined.Employing uniquely identifiable optical targets 102 b and 102 b′ solvesthis potential problem.)

In FIG. 1B, the particular hand-held surgical tool 104 is appreciated tobe a generic syringe 104 b. The syringe 104 b has a working end 106 bthat is understood by one skilled in the art to be the tip of a needle(located at the distal end of the syringe 104 b that is used to puncturea patient for injecting medicine or drawing tissue samples. Prior to useof the syringe 104 b during the surgical procedure, one or moreidentifiable optical targets 102 a and/or 102 b are placed at knownlocations (or are fabricated at the known locations) of the surface ofthe syringe 104 b in the various preferred embodiments. Preferably, eachof the optical targets 102 will be uniquely identifiable. (It isappreciated that one or more optical targets 102 a, and/or three or moreoptical targets 102 b, might be used in other embodiments.)

Additionally, some embodiments may be configured to determine therelative distance between the optical targets 102 b, 102 b′ during thesurgical procedure. For example, the optical target 102 b may be locatedon a plunger 110 of the syringe 104 b. The second optical target 102 b′may be located on the barrel 112 of the syringe 104 b The change indistance between the optical targets 102 b, 102 b′ during the course ofthe surgical procedure may then be used to precisely compute a traveldistance of the barrel seal 114 that is at the distal end of the plunger110. Since the volume of the barrel 112 is known and the volumeinformation is stored as part of the syringe model data, the volume ofmedicine injected out from the syringe 104 b and/or the amount of tissuecollected within the syringe 104 b may be precisely determined based onthe determined travel distance of the barrel seal 114. This informationmay be presented to the practitioner on a real-time basis during thesurgical procedure.

In FIG. 1C, the particular hand-held surgical tool 104 is appreciated tobe a generic surgical scissor 104 c. The surgical scissor 104 c has afirst working end 106 c and a second working end 106 c′ (located at thedistal end of the surgical scissor 104 c) that is understood by oneskilled in the art to be the cutting end that is used to cut tissueduring a surgical procedure. Prior to use of the surgical scissor 104 cduring the surgical procedure, a plurality of identifiable opticaltargets 102 are placed at known locations (or are fabricated at theknown locations) of the surface of the surgical scissor 104 c in thevarious preferred embodiments. Preferably, each of the optical targets102 will be uniquely identifiable.

One skilled in the art appreciates that for precise determination of thelocation of the cutting edges or the surgical scissor 104 c, the preciselocation of each working end 106 c. 108 c′ must be individuallydetermined. To enable this determination of the precise location andorientation of the surgical scissor 104 c and the working ends 106 c,106 c′ during the surgical procedure, each of the two first surgicaltool members 108 c and 108 c′ of the surgical scissor 104 c must eachhave their own optical targets 102. In the non-limiting exampleembodiment illustrated in FIG. 1C, the first surgical tool member 108 cincludes a first optical target 102 b and a second optical target 102b′. The second surgical tool member 108 c′ includes a third opticaltarget 102 b″ and a fourth optical target 102 b′″. The determinedlocation of the optical targets 102 b″, 102 b′″ may be used to determinethe precise location and orientation of the working end 106 c.Similarly, the determined location of the optical targets 102 b, 102 b′may be used to determine the precise location and orientation of theworking end 106 c′. Alternatively, or additionally, the relativeseparation between the optical targets 102 b and 102 b″, and/or theoptical targets 102 b′ and 102 b′″, may be used in the computations.

Alternatively, or additionally, a plurality of optical targets 102 a(not shown) may be used to track the members 108 c, 108 c′. Threeoptical targets 102 b may be used (see FIG. 1D, for example).

In practice, the surgical scissor 104 c may be closed at some pointduring the surgical procedure wherein the working ends 106 c, 106 c′ areadjacent to and/or are in contact with each other. At another pointduring the surgical procedure, the surgical scissor 104 c may be openedsuch that the working ends 106 c, 106 c′ are separated from each otherby a determinable distance so that the practitioner can insert theworking ends 106 c. 106 c′ around the tissue that is to be cut. Further,as the practitioner is cutting the tissue of interest, the working ends106 c. 106 c′ are moving towards each other in a cutting motion.Accordingly, determination of the precise location and orientation ofthe surgical scissor 104 c, and in particular the working ends 106 c,106 c′, is determinable in real-time or near real-time.

In FIG. 1D, the particular hand-held surgical tool 104 is appreciated tobe a generic surgical clamp 104 d. The surgical clamp 104 d has a firstworking end 106 d and a second working end 106 d′ (located at the distalend of the surgical clamp 104 d) that is understood by one skilled inthe art to be clamps that are used to damp tissue during a surgicalprocedure. Prior to use of the surgical clamp 104 d during the surgicalprocedure, a plurality of identifiable optical targets 102 are placed atknown locations (or are fabricated at the known locations) or thesurface of the surgical clamp 104 d in the various preferredembodiments. Preferably, each of the optical targets 102 will beuniquely identifiable.

Similar to the above-described surgical scissor 104 c, one skilled inthe art appreciates that for precise determination of the location ofthe clamping ends of the surgical clamp 104 d, the precise location ofeach working end 106 d, 106 d′ must be individually determined. Forexample, the surgical clamp 104 d may be opened such that the workingends 106 d, 106 d′ are separated from each other by a determinabledistance so that the practitioner can insert the working ends 106 d. 106d′ around the tissue that is to be clamped. At another point during thesurgical procedure, the surgical clamp 104 d may be later closed duringthe surgical procedure wherein the working ends 106 d, 106 d′ areadjacent to and/or are in contact with each other to clamp the tissue orinterest. Further, as the practitioner is clamping the tissue ofinterest, the working ends 106 c, 106 c′ are moving towards each otherin a clamping motion. Accordingly, determination of the precise locationand orientation of the surgical clamp 104 d, and in particular theworking ends 106 d, 106 d′, is determinable in real-time or nearreal-time.

In the non-limiting example of the surgical clamp 104 d, to enable thedetermination of the precise location and orientation of the surgicalclamp 104 d and the working ends 106 d, 106 d′ during the surgicalprocedure, each of the two surgical tool members 108 d and 108 d′ of thesurgical clamp 104 d are conceptually illustrated as having their ownoptical target 102 a. In the non-limiting example embodiment illustratedin FIG. 1D, the first surgical tool member 108 d includes a firstoptical target 102 a. The second surgical tool member 108 d′ includes asecond optical target 102 b′. Once the precise location and orientationof each arm 108 d, 108 d′ is determined, the precise location andorientation of the surgical clamp 104 d and its associated working ends106 d, 106 d′ can be determined by the processor system 202.

Alternatively, or additionally, to enable the determination of theprecise location and orientation of the surgical clamp 104 d and theworking ends 106 d, 106 d′ during the surgical procedure, each of thetwo surgical tool members 108 d and 108 d′ of surgical damp 104 d areconceptually illustrated as having their own optical targets 102 b and102 b′, respectively. A third optical target 102 b″ is located at thehinge of the surgical clamp 104 d (where the first surgical tool member108 d Is hinge-ably coupled to the second surgical tool member 108 d″).Here, the precise location and orientation of the first surgical toolmember 108 d (first arm 108 d) can be determined based on the determinedlocation of the first optical target 102 b and the hinge optical target102 b″. Similarly, the precise location and orientation of the secondsurgical tool member 108 d′ can be determined based on the determinedlocation of the first optical target 102 b′ and the hinge optical target102 b″. Once the precise location and orientation of each surgical toolmember 108 d, 108 d′ is determined, the precise location and orientationof the surgical clamp 104 d and its associated working ends 106 d, 106d′ can be determined by the processor system 202 in real-time or nearreal-time.

As noted herein, a hand-held surgical tool 104 may have only one type ofdetectable target 102 or a plurality of different types of detectabletargets 102. For example, the non-limiting surgical clamp 104 d uses twodifferent types of optical targets 102 a and 102 b. Other embodiments ofthe surgical clamp 104 d may use only one of the types of the opticaltargets 102 (and/or may use other types of optical targets 102). Use ofdifferent types of optical targets 102 enables the same hand-heldsurgical tool 104 to be used with different embodiments of the hand-heldsurgical tool tracking system 100 that employ different image processingalgorithms modules 212 that are configured to identify a particular typeof optical target 102. This feature of detectable targets 102 may beused on any hand-held surgical tool 104.

In FIG. 1E, the particular hand-held surgical tool 104 is appreciated tobe a generic surgical tweezer 104 e. The surgical tweezer 104 e has afirst working end 106 e and a second working end 106 e′ (located at thedistal end of the surgical tweezer 104 e) that is understood by oneskilled in the art to be the grasping tool end that is used to grasptissue during a surgical procedure. Prior to use of the surgical tweezer104 e during the surgical procedure, a plurality of identifiable opticaltargets 102 are placed at known locations (or are fabricated at theknown locations) of the surface of the surgical clamp 104 d in thevarious preferred embodiments. Preferably, each of the optical targets102 will be uniquely identifiable.

In the non-limiting example of the surgical tweezer 104 e, to enable thedetermination of the precise location and orientation of the surgicaltweezer 104 e and the working ends 106 e, 108 e′ during the surgicalprocedure, each of the two surgical tool members 108 e and 108 e′ of thesurgical tweezer 104 e are conceptually illustrated as having their ownoptical target 102 a, 102 a′, respectively. Once the precise locationand orientation of each surgical tool member 108 e. 108 e′ isdetermined, the precise location and orientation of the surgical tweezer104 e and its associated working ends 10 e, 106 e′ can be determined bythe processor system 202.

One skilled in the arts appreciates that there are a variety ofdifferent types of hand-held surgical tools that might be used by apractitioner during a surgical procedure. Accordingly, the examplehand-held surgical tools 104 of FIGS. 1A-1E are intended to representonly a sampling of possible hand-held surgical tools that may be trackedusing detectable targets 102. Further, some hand-held surgical tools mayhave working ends locate at both the proximal and distal ends of thehand-held surgical tool (such as encountered in the dentistry arts). Allsuch manually manipulated hand-held surgical tools 104 now known orlater developed are intended to be within the scope or this disclosureand to be protected by the accompanying claims.

Some embodiments may employ a plurality of different detectable targets102 a, 102 b located on other surfaces of the hand-held surgical tool104. For example, detectable targets 102 a, 102 b may be located on theopposing surface of the scalpel 104 a and or on the side surfaces of thescalpel 104 a. For example, multiple detectable optical targets 102 a.102 b may be on the same surface of the hand-held surgical tool 104 incase the practitioner's hand or another object blocks a view of thehand-held surgical tool 104 by the image capture device 206 during thesurgical procedure. Accordingly, a sufficient number of detectableoptical target(s) 102 a. 102 b will always be discernable in an imagethat is being captured by the image capture device 206 (FIG. 2 ).Therefore, the practitioner does not need to worry about how they aregrasping the hand-held surgical tool 104 during the surgical procedure.

As noted herein, the precise location and orientation of the working end106 of any hand-held surgical tool 104 is precisely determined based ona determined location of the one or more detectable targets 102. Oneskilled in the arts understands how such location and orientationcalculations are performed to identify the precise location andorientation of the hand-held surgical tool 104 in a known 3D space basedon the determined locations of detectable targets 102, such as theexample optical targets 102 a, 102 b. Accordingly, such geometrycalculations are not described herein for brevity.

Prior to performing a surgical procedure using any one of the availablehand-held surgical tools 104, the precise location of each detectabletarget 102 on the surface of the hand-held surgical tool 104 is known.The known location of each detectable target 102 may be based on designspecifications. During manufacture, each of the detectable targets 102are fabricated at their precisely known locations and/or orientations.An unexpected advantage of incorporating the detectable target(s) 102 aspart of a manufactured hand-held surgical tool 104 is that hundreds,even thousands, of like hand-held surgical tools 104 may be manufacturedand distributed to different sites. If the site of the surgicalprocedure has an embodiment of the hand-held surgical tool trackingsystem 100, then the precise location and orientation of the hand-heldsurgical tool 104 can be determined during the surgical procedure.However, if the site does not have an embodiments of the hand-heldsurgical tool tracking system 100, the hand-held surgical tool 104 stillmay be used by the practitioner to perform the surgical procedure in aconventional manner.

Alternatively, or additionally, the detectable target(s) 102 may beplaced onto the surface of the hand-held surgical tool 104 after itsmanufacture. Here, each detectable target 102 is secured to the surfaceof the hand-held surgical tool 104 at a precisely known location and/ororientation (using an adhesive, by painting, or the like). Preferably,the detectable target(s) 102 are secured to many of the like hand-heldsurgical tools 104 at identical locations prior to distribution to thedifferent sites where a surgical procedure is to be performed.

An unexpected advantage of distributing many like hand-held surgicaltools 104, each with their detectable targets 102 at the identicallysame location and/or orientation on the hand-held surgical tool 104, isthat a single model of that particular hand-held surgical tool 104 maybe generated and stored. The hand-held surgical tool model data may besaved locally at the surgical instrument model database 216 (FIG. 2 )and/or remotely in a remote database.

For example, thousands of scalpels 104 a may be manufactured anddistributed to many different surgical sites (e.g., hospitals, clinics,etc.). If the surgical site where the surgical procedure is beingperformed is provisioned with an embodiment of the hand-held surgicaltool tracking system 100, then the practitioner (or their assistant) canuse any of the available scalpels 104 a during the surgical procedure.Further, different types of hand-held surgical tools 104 may be used bythe practitioner during the surgical procedure since each different typeof hand-held surgical tool 104 is readily identifiable by embodiments ofthe hand-held surgical tool tracking system 100.

Yet another unexpected advantage is that any particular type ofhand-held surgical tool 104 with precisely located detectable targets102 is that different manufacturers of the particular type of hand-heldsurgical tool 104 may provide their proprietary tool 104 to thepractitioners for various surgical procedures. If the manufacturersproduce identical hand-held surgical tools 104, a single model mayrepresent the hand-held surgical tool 104 that has been distributed bydifferent manufacturers. In the event that there are differences betweena particular type of hand-held surgical tool 104 being produced bydifferent manufacturers, then a hand-held surgical tool model may begenerated and saved for each particular manufacturer.

Alternatively, or additionally, the detectable targets 102 may besecured to a hand-held surgical tool 104 of interest (using a suitableadhesive or paint) by the practitioner or another party prior to use.For example, one or more images of the hand-held surgical tool 104 arecaptured after manual placement of the optical targets 102 during ahand-held surgical tool calibration process. Embodiments of thehand-held surgical tool tracking system 100 then analyze the acquiredimage data to identify the precise location and/or orientation of theoptical target(s) 102, and the location of the working end(s) 106 of thehand-held surgical tool 104. Then, embodiments of the hand-held surgicaltool tracking system 100 may generate a calibrated model of thephotographed hand-held surgical tool 104. An unexpected advantage ofthis embodiment is that any hand-held surgical tool 104 of interest maybe fitted with one or more optical targets 102, and then may be usedduring the surgical procedure.

In the conceptual illustration of FIG. 2 , the patient 200 is laying ontheir stomach on an operating table or the like. Hypothetically, assumethat the practitioner (not shown) is intending to insert the needle 106b (the working end) of the syringe 104 b into an intended locationwithin the patient's spine (not shown). Although the precise locationand orientation of the syringe 104 b in the 3D space is determinable, itis not possible to know the exact location of the patient's tissue ofinterest (here, the patient's spine) because of the patient's coveringskin.

Accordingly, a calibration of patient 200 is required before initiationof the surgical procedure. In the conceptual example illustrated in FIG.2 , at least one detectable target 228 is placed on the patient 200 orat another suitable location proximate to the patient 208. If an opticaltarget 228 is used, an image capture device 206 acquires an image of theoptical target 228, and then determines the precise location andorientation of the optical target 228. Since the optical target 228 islocated at a known location with respect to the patient (proximate to),the location of the tissue of interest may be computed based oninformation that is known about the patient 200. That is, a locationrelationship between a tissue of interest of the patient 200 and thepredefined location of the detectable target 228 is known. Once thelocation of the tissue of interest is known, then the relative locationrelationship between the tissue and the hand-held surgical tool 104 canbe determined during the course of the surgical procedure. This patienttissue calibration approach may be adequate if the tissue of interest isvisible on the surface of the patient 200, and/or if a high degree ofprecision is not needed. For example, if the tissue of interest is amote or the like on the skin of the patient 200, this patient tissuecalibration technique may be adequate. Location of the tissue ofinterest may be enhanced by embodiments of the hand-held surgical tooltracking system 100 using object recognition techniques. For example, animage of the skin of the patient may be analyzed to identify and locatethe mole on the patient's skin.

However, one skilled in the art appreciates that the above-describedpatient tissue calibration may not be sufficient to identify the preciselocation and orientation of other types of tissue of interest,particularly if the tissue of interest is internal and/or If the tissueis especially sensitive. For example, if the needle 106 b is to be usedto puncture the spine of the patient 200, a very precise patientcalibration is required.

In some embodiments, patient calibration is performed using anotherdevice or with an embodiment of the hand-held surgical tool trackingsystem 100. For example, but not limited to, the optical target 228 maybe placed on the patient 200, preferably in proximity to the location ofthe tissue of interest. Then, an ultrasonagraphic system in accordancewith U.S. Pat. Nos. 9,675,321 and 9,713,508, or an embodiment of thehand-held surgical tool tracking system 100 modified to incorporate thefeatures of the above-Identified ultrasonographic system, may be used toacquire one or more ultrasonic scans of the tissue of interest. Sincethe precise location and orientation of the ultrasonic sonic scanner isknown (since it also has one or more discernable optical targets 102 onits surface), the precise location and orientation of the patient'stissue of interest with respect to the patient's detectable target 228is determinable. Once patient tissue calibration has been completed, thesurgical procedure may be initiated.

Yet another example of patient tissue calibration, the detectable target228 may have a portion that is optically detectable, and a portion thatis detectable in Xray images. CT images, fluoroscopy images, or thelike. For example, one or more metal beads or the like may be at a knownlocations on the detectable target 102. Prior to the initiation of thesurgical procedure, the detectable target 228 may be secured to thepatient 200. Then, one or more images may be acquired of the patient200. Since the tissue of interest and the detectable target 228 arediscernable in the acquired image(s), then the precise location andorientation of the optical target 228 to the tissue of interest can bedetermined by the processor system 202.

As yet another example of patient tissue calibration, an MRI and/or CTsystem may be used to scan the patient. Since the tissue of interest andthe optical target 228 are discernable in the acquired MRI and/or CTdata, then the precise location and orientation of the detectable target228 to the tissue of interest can be determined by the processor system202.

Alternatively, or additionally, the patient tissue calibration processmay occur on a real-time basis during the surgical procedure process.For example, sonogram images, Xray images, or the like can be acquiredof the patient 200 (and their detectable target 228) during the surgicalprocedure. Embodiments of the hand-held surgical tool tracking system100 may then calibrate in real-time. This calibration approach may beparticularly desirable if the tissue of interest is moving during thesurgical procedure. For example, the tissue of interest may be a beatingheart of the patient 200 or the lungs of the patient 200 who isbreathing during the surgical procedure.

Any suitable system for acquiring patient tissue calibration informationduring the surgical procedure is intended to be included within thescope of this disclosure and to be protected by the accompanying claims.Such patient tissue calibration can be acquired prior to, and/or during,the surgical procedure.

In practice, during the surgical procedure, the 2D or 3D model of thetissue of interest (an organ, bone, etc.) is retrieved from the tissuemodel data base 218. The retrieved tissue model of the patient 200undergoing the current surgical procedure has been previously generatedfrom an examination of the patient 200 prior to the current surgicalprocedure, and then stored in the tissue model data base 218. As thepractitioner begins the surgical procedure, the hand-held surgical tooltracking system 100 determines the precise location and orientation ofthe hand-held surgical tool 104 in 3D space.

In an example embodiment, the image capture device 206 captures imagesin real tine that includes the optical targets 102 a, 102 b on thehand-held surgical tool 104 and the optical target 228 on the patient200. The clock 222 may optionally provide time stamps to acquiredimages. The captured image data is then communicated from the imagecapture device 206 to the target tracker unit 208.

In some embodiments, a plurality of image capture devices 206 may beused to capture camera images from different viewpoints in asynchronized fashion. Here, the multiple image capture devices 206provide concurrently captured camera images with the same time stamp.

The target tracker unit 208, for each acquired image, Identifies the oneor more optical targets 102 on the hand-held surgical tool 104 and theoptical target 228 that have been placed on the surface or the body ofthe patient 200. The target tracker unit 208 then computes or determinesthe precise location and orientation of the optical targets 202 relativeto the optical target 228 in 3D space for the indexed time. In someembodiments, the image data is analyzed to identify the particularhand-held surgical tool 104 that is currently being used by thepractitioner.

In some embodiments, one or more of the detectable targets 102, 228 maynot be optically detectable (since they may not be reflecting light inthe visible spectrum). Another detecting device 206 is used to acquireimage data from the detectable targets 102, 228 (that emit energy fromother non-visible spectrums) to detect the precise location andorientation of the detectable targets 102. Then the target tracker unit208 can determine the precise location and orientation of the detectabletargets 202 relative to the detectable target 228 in the 3D space.

The image registration module 210 then receives the location andorientation information for the detectable targets 202 and 228. Theimage registration module 210 also retrieves the model data from thesurgical tool model database 216 for the particular hand-held surgicaltool 104 that is being used by the practitioner (and that has beenoptionally identified and/or verified in the captured images). Theposition of the hand-held surgical tool 104, and the position of theworking end 106 of the hand-held surgical tool 104, is determined basedupon the identified relative location of the detectable targets 102 and228 in the acquired camera image and based on correlation with theretrieved model data of the corresponding hand-held surgical tool 104.Information corresponding to the precise location and orientation of thehand-held surgical tool 104 and its working end 106 in the 3D space isthen communicated to the image processing algorithm module 212.

The image processing algorithm module 212 retrieves the previouslygenerated and saved tissue model data of the tissue of interest of thepatient 200 from the tissue model database 218. Preferably, the tissuemodel is based on the patient's tissue of interest during a previoustissue model generation process. When the tissue of interest is static(not moving, such as a bone or the like) during the surgical procedure,the tissue model data can be retrieved when the surgical procedure isinitiated. Since the relative location and orientation of the hand-heldsurgical tool 104 (and its working end 106) relative to the detectabletarget 228 in the 3D space has been determined and, since the locationof the tissue of interest relative to the detectable target 228 has beendetermined (during the patient calibration), the image processingalgorithm module 212 may generate a real-time composite image thatincludes both an image of the hand-held surgical tool 104 and an imageof the tissue of interest (as represented by the tissue model).

Data corresponding to the generated real-time composite image iscommunicated to the 3D/2D visualization module 214. The 3D/2Dvisualization module 214 then generates image data for presentation on a3D display and/or a 2D display. In embodiments with the 3D/2D display224, the composite image showing the precise location and orientation ofthe hand-held surgical tool 104, and in particular the working end 106of the hand-held surgical tool 104 with respect to the tissue ofinterest (represented by the predetermined tissue model of the tissue ofinterest) is then rendered and presented on the display 224. Thepresented composite image may be either a 3D image or a 2D imagedepending upon the characteristics of the particular display 224.

In some embodiments of the hand-held surgical tool tracking system 100,the practitioner or another user may provide input to the 3D/2Dvisualization module 214 to adjust presentation of the composite imagevia the user input device 220. Any suitable type of under inputdevice(s) 220 may be used in the various embodiments. In response toreceiving the user input, the 3D/2D visualization module 214 may modifypresentation of the composite image in accordance with the user input.

In some situations, the user input may be a request to zoom in on aparticular location or region in the composite image. For example, thepractitioner may wish to see a closeup image that presents a magnifiedimage of the working end 106 of the hand-held surgical tool 104 and thetissue that is in proximity to the working end 106. Accordingly, thepractitioner may be able to discern in greater detail and accuracyprecisely where the working end 106 of the hand-held surgical tool 104is currently at with respect to the tissue of interest.

As another exemplary situation, since the model of the tissue ofinterest is in 3D space and the precise location and orientation of theworking end 106 of the hand-held surgical tool 104 is known in the 3Dspace, the practitioner or another user may wish to rotate the presentedview of the composite image. For example, the practitioner may wish toexamine a side view or a bottom of the composite image. The 3D/2Dvisualization module 214 may rotate the composite image so that thepractitioner or another user is then presented the side view or thebottom view on the display 224.

Alternatively, or additionally, the composite image Information can becommunicated to the remote rendering and display system 204. Thecomposite information that is rendered to present an image of theprecise location and orientation of the hand-held surgical tool 104 andthe tissue of interest can then be presented on a display of the remoterendering and display system 204. For example, the remote rendering anddisplay system 204 may be a virtual reality system that employs a headset display device.

The practitioner or other user viewing the composite image on thedisplay of the remote rendering and display system 204 may wish toadjust the view of the composite image. In some embodiments, Informationcorresponding to the user's view request is communicated from the remoterendering and display system 204 to the 3D/2D visualization module 214.The 3D/2D visualization module 214 adjusts the composite view and thencommunicates the adjusted composite view to the remote rendering anddisplay system 204 for presentation on the display. Alternatively, oradditionally, the composite image data that is communicated to theremote rendering and display system 204 may include 3D model data ofboth the hand-held surgical tool 104 and the tissue of interest. Thepresented image on the display of the remote rendering and displaysystem 204 may then be adjusted and presented by the remote renderingand display system 204.

It is appreciated by one skilled in the art that the processes ofrendering, manipulating and presenting images based on 2D and 3Dmodelling technologies is well known in the arts. Such technologies arenot described herein for brevity. All such technologies now known orlater developed are considered to be within the scope of this disclosureand to be protected by the accompanying claims.

FIG. 3 is a conceptual illustration of a presented composite image 302showing the relative location of a spine 304 or a patient 200 and ahand-held surgical tool 104. Here, the spine 304 is an example of astatic tissue of interest that is not moving during the surgicalprocedure. The static tissue of interest might be other types of organs,such as a liver, a brain, a muscle or another bone. During some surgicalprocedures, a portion of the patient's body may be secured using asuitable securing device, such as, but not limited to, a clamp, a strap,or the like, that immobilizes the tissue of interest during the surgicalprocedure.

In this simplified conceptual example, the practitioner is inserting theneedle 106 b of the syringe 104 b in between two vertebrae of thepatient's spine, represented by the rendered spine model 304. Oneskilled in the art appreciates the delicate nature of this surgicalprocedure, and appreciates the importance of precisely locating andorienting the tip 106 b of the syringe 104 b for an injection or fortissue sampling. Typically, the practitioner is only viewing the skinsurface of the patient 200, as illustrated in FIG. 2 . Embodiments ofthe hand-held surgical tool tracking system 100 determine the preciselocation and orientation of the needle tip 106 b relative to thepatient's spine (based on the previously generated tissue model 304 ofthe patient's spine), and present a composite image 302 on a displaythat is viewable by the practitioner or another party. Further, movementof the needle tip 106 b can be presented in the presented compositeimage 302 in real-time or near real-time. Here, the practitioner is ableto view the needle tip 106 b and the vertebrae of the spine as theneedle tip 106 b is puncturing the patient 200. Once the practitioner issatisfied with the precise location and orientation of the needle tip106 b, the practitioner can operate the syringe 104 b to inject medicineinto the spine or to retrieve a tissue sample from the spine.

Non-stationary tissue (moving tissue) presents a more complex issue ofpresenting the composite image of the tissue of interest and thehand-held surgical tool 104 on a real-time basis. For example, thetissue of interest may be a beating heart of the patient. Here, thepreviously generated tissue model of the tissue of interest may alsoinclude tissue movement characteristics. For example, if a heart is thetissue of interest, then the generated 3D model must have sufficientdata to represent the beating of the heart. That is, the moving portionsof the patient's heart must be visible in the presented composite imagethat represents the patient's beating heart and the precise location andorientation of the hand-held surgical tool 104.

Generation of 2D and 3D dynamic models representing moving objects iswell understood in the arts. For brevity, such dynamic model generationis not described in detail herein. All such dynamic model generationsystems and processes now known or later developed are intended to beincluded within the scope of this disclosure and to be protected by theaccompanying claims.

Embodiments of the hand-held surgical tool tracking system 100 areconfigured to synchronize presentation of movement of the dynamic tissuemodel with actual movement of the patient's tissue in real-time. Forexample, the presented composite image will show a model of thepatient's beating heart that corresponds to the actual beating of theheart. Similarly, other moving tissues will be synchronized with theirdynamic tissue models.

In the various embodiments, a detector 232 that is configured to detectmovement or tissue may be used to monitor movement of the tissue ofinterest during the surgical procedure. The detector 232 iscommunicatively coupled to the processor system 202 via a suitablewireless or wire-based connector. Tissue movement information iscommunicated from the detector 232 to the image processing algorithmmodule 212. The image processing algorithm module 212 synchronizesmovement of the dynamic tissue model with the movement of the patient'stissue on a real-time or near real-time basis. Accordingly, thepresented composite image accurately represents the movement of thetissue of interest.

In the situation where the tissue of interest is a beating heart, anaudio detector such as a stethoscope or the like may be used to detectthe beating of the patient's heart. As the heart beats are detected,information corresponding to the detected heart beating is communicatedto the processor system 202. The movement of the dynamic modelcorresponding to the patient's heart may then be synchronized to thereceived heart beating information.

Detection of moving tissue is well known in the arts and is notdescribed herein for brevity. Further, synchronization of the movementof dynamic models based on actual movement of the modelled object (here,the patient's tissue) is well known in the arts and is not describedherein for brevity. All such tissue movement detection and/or dynamicmodel generation systems and processes now known or later developed areintended to be included within the scope of this disclosure and to beprotected by the accompanying claims.

FIG. 4 is a conceptual diagram of an embodiment of the hand-heldsurgical tool tracking system 100 and a robotic operation system 402cooperatively working together to generate a composite image thatincludes robotic tools 404 and hand-held surgical tools 104. In thissimplified hypothetical example, the patient 200 is laying on a surgicaltable 406 during a surgical procedure. A legacy robotic operation system402 is manipulating one or more robotic tools 404 in accordance withinstructions received from the robot controller 408 that are specifiedby the practitioner 410.

During the surgical procedure, the robotic operation system 402determines the precise location and orientation of each of the robotictools 404 as is known in the art of robotic tools. A graphical image ofthe tissue operating area 412 and the robotic tools 404 may be presentedon a display 414 being viewed by the practitioner 410. Depending uponthe robotic system, a 2D image of the operating area 412 and the robotictools 404 may be presented on the display 414. Alternatively, oradditionally, a 2D or 3D model of the tissue and the robotic tools 404may be presented on the display 414.

On occasion, an assistant 416 may be required to assist or intervene inthe surgical procedure by using one or more hand-held surgical tools104. The robotic operation system 402, at best, can only obtain 2D imageinformation showing the intervening hand-held surgical tools 104 as itare being used. It is not possible for the robotic operation system 402to determine precise location and orientation of the interveninghand-held surgical tool 104 that is being used by the assistant.

However, the precise location and orientation of the interveninghand-held surgical tool 104 can be determined by embodiments of thehand-held surgical tool tracking system 100. Here, the image capturedevice 206 is positioned so as to capture images of the opticallydetectable targets 102 on the surface of the hand-held surgical tool104. The 3D space known by the hand-held surgical tool tracking system100 is the same as the 3D space known by the robotic operation system402. Accordingly, image information presenting the precise location andorientation of the intervening hand-held surgical tool 104 can begenerated by the hand-held surgical tool tracking system 100. Thisinformation may be communicated to the remote rendering and displaysystem 204 for presentation of the display 214 along with concurrentpresentation of graphical information generated by the robotic operationsystem 402. Accordingly, the practitioner 410 may concurrently view therobotic tools 404 being controlled by the robotic operation system 402and the intervening hand-held surgical tool 104 being used by theassistant 416. If a 2D or 3D model of the tissue of interest in theoperating area 412 is being presented on the display 414 (by either thehand-held surgical tool tracking system 100 or the robotic operationsystem 402), then the practitioner 410 is able to concurrently view theintervening hand-held surgical tool 104 and the robotic tools 404 inrelation to the presented tissue model.

One skilled in the art appreciates that numerous robotic operationsystems 402 are now known or will be developed in the future. Suchrobotic operation systems 402 may graphically present variousinformation on a display 414 to the practitioner 410 who is operatingthe robotic operation system 402. These numerous robotic operationsystems 402 are not described in detail herein for brevity. Further,integration of image information from multiple image generation sourcesinto a single image is well known in the arts, and is not describedherein in detail for brevity. Here, image informant generated byembodiments of the hand-held surgical tool tracking system 100 isintegrated with image information generated by the robotic operationsystem 402 so that the practitioner 410 can appreciate the preciselocation and orientation of any intervening hand-held surgical tools 104used during the surgical procedure. All such forms of robotic operationsystems 402 know known or later developed, and all techniques ofintegrating image information know known or later developed, areconsidered to be within the scope of this disclosure and to be protectedby the accompanying claims.

In some embodiments of the hand-held surgical tool tracking system 100,Information corresponding to the generated and presented compositeimages (showing the tissue model and the hand-held surgical tools 104)may be saved into a local memory medium, such as the example surgicalprocedure history 230 (FIG. 2 ), and/or saved into remote memory medium(not shown). Preferably, the composite image information is timestamped. Accordingly, an interested party may later review and analyzethe surgical procedure by viewing the saved composite images. The storedcomposite images may be individually viewed, or may be viewed as avideo.

Determining precise location and orientation of hand-held surgical toolsand other objects takes a discernable amount of time and is verycomputationally intensive for legacy object recognitions systems. Suchlegacy object recognition systems may not have sufficient computationalcapacity (processor system speed and or bandwidth) to determine preciselocation and orientation of an object in real-time or near real-timebased on object recognition techniques alone. Embodiments of thehand-held surgical tool tracking system 100 address this problem bydetermining precise location and orientation of one or more detectableoptical targets 102, and then correlating the detected optical targets102 with known location of the optical targets of on a known model ofthe hand-held surgical tool 104.

As noted herein, detectable targets 102 may be active, such as byemitting infrared signals to the optical target, or passive, such asincluding retro-reflective markers affixed to some interaction device.Such active or passive detectable targets 102 are generically describedherein as detectable targets for brevity, though such detectable targets102 may not be optically detectable by an image capture device. Rather,the active or passive detectable targets 102 are detectable usinganother detecting device 206 or system 206.

It should be emphasized that the above-described embodiments of thehand-held surgical tool tracking system 100 are merely possible examplesof implementations of the invention. Many variations and modificationsmay be made to the above-described embodiments. All such modificationsand variations are intended to be included herein within the scope ofthis disclosure and protected by the following claims.

Furthermore, the disclosure above encompasses multiple distinctinventions with independent utility. While each of these inventions hasbeen disclosed in a particular form, the specific embodiments disclosedand illustrated above are not to be considered in a limiting sense asnumerous variations are possible. The subject matter of the inventionsincludes all novel and non-obvious combinations and subcombinations ofthe various elements, features, functions and/or properties disclosedabove and inherent to those skilled in the art pertaining to suchinventions. Where the disclosure or subsequently filed claims recite “a”element, “a first” element, or any such equivalent term, the disclosureor claims should be understood to incorporate one or more such elements,neither requiring nor excluding two or more such elements.

Applicant(s) reserves the right to submit claims directed tocombinations and subcombinations of the disclosed inventions that arebelieved to be novel and non-obvious. Inventions embodied in othercombinations and subcombinations of features, functions, elements and/orproperties may be claimed through amendment of those claims orpresentation of new claims in the present application or in a relatedapplication. Such amended or new claims, whether they are directed tothe same invention or a different invention and whether they aredifferent, broader, narrower, or equal in scope to the original claims,are to be considered within the subject matter of the inventionsdescribed herein.

The inventions described in this application may be made by a variety ofindustrial processes, including by various mechanical, electrical, andpneumatic assembly techniques. Further, the inventions described hereinmay be used in industrial contexts, Including surgical procedureendeavors.

Therefore, having thus described the invention, at least the followingis claimed:
 1. A surgical method, comprising: capturing using an imagecapture device, image data that includes both a first detectable targeton a hand-held surgical tool and a second detectable target proximate toa patient, wherein the first detectable target is located on thehand-held surgical tool at a first predefined target location, whereinthe second detectable target is located on the patient at a secondpredefined target location, and wherein a location relationship betweena tissue of interest of the patient and the second predefined locationof the second detectable target is known; determining a location of thefirst detectable target and an orientation of the first detectabletarget in a three dimensional (3D) spaced based on the image data;retrieving hand-held surgical tool model data that represents thehand-held surgical tool, wherein the hand-held surgical tool model datacomprises first detectable target location model data that correspondsto the predefined location of the first detectable target; determining acurrent location of the hand-held surgical tool and a currentorientation of the hand-held surgical tool in the 3D space based on thedetermined location of the first detectable target and the determinedorientation of the first detectable target; determining a location ofthe second detectable target in the 3D space based on the image data;determining a location of the tissue of interest in the 3D space basedon the determined location of the second detectable target on thepatient; and accessing tissue model data defining a tissue modelcorresponding to the tissue of interest of the patient, wherein thetissue model was previously generated from an examination of the patientprior to the current surgical procedure.
 2. The surgical method of claim1, wherein the tissue model data was previously generated based on aseries of acquired time indexed sonogram images.
 3. The surgical methodof claim 1, further comprising: generating a composite image, whereinthe generated composite image concurrently presents a first graphicalrepresentation of the tissue of interest of the patient based on thetissue model and a second graphical representation of the hand-heldsurgical tool; and presenting the generated composite image on adisplay, wherein a surgical team member viewing the display intuitivelyunderstands the current location of the hand-held surgical tool and thecurrent orientation of the hand-held surgical tool with respect to thetissue of interest of the patient.
 4. The surgical method of claim 3,wherein the hand-held surgical tool model data includes working endlocation model data that corresponds to a location of a working end ofthe hand-held surgical tool, and wherein the working end of thehand-held surgical tool is disposed on a distal end of the hand-heldsurgical tool, the method further comprising: determining a location ofthe working end of the hand-held surgical tool and an orientation of theworking end of the hand-held surgical tool based on the hand-heldsurgical tool model data, and wherein the surgical team member viewingthe display intuitively understands the current location of the workingend of the hand-held surgical tool and the current orientation of theworking end of the hand-held surgical tool with respect to the tissue ofinterest of the patient.
 5. The surgical method of claim 3, wherein thecaptured image data is one of a plurality of time sequenced capturedimages of a video stream captured by the image capture device, themethod further comprising: continuously updating the generated imagethat is presented on the display using currently acquired image datathat is received in the video stream, wherein the surgical team memberviewing the display intuitively understands the current location of thehand-held surgical tool and the current orientation of the hand-heldsurgical tool with respect to the tissue of interest of the patient inreal-time.
 6. The surgical method of claim 1, wherein the firstdetectable target is a first optical target, wherein the seconddetectable target is a second optical target, and wherein the imagecapture device is a camera.
 7. The surgical method of claim 1, wherein athird detectable target is located on the hand-held surgical tool at asecond predefined target location that is different from the firstpredefined target location, wherein the hand-held surgical tool modeldata comprises third detectable target location model data thatcorresponds to the predefined location of the third detectable target,the method further comprising: determining a location of the thirddetectable target the 3D space based on the image data, and based on thehand-held surgical tool model data, wherein determining the currentlocation of the hand-held surgical tool and the current orientation ofthe hand-held surgical tool in the 3D space is based on the determinedlocation of the first detectable target and the determined location ofthe third detectable target.
 8. The surgical method of claim 1, whereinthe hand-held surgical tool is a surgical scalpel with a surgical toolmember that is a scalpel blade that is located at the distal end of thesurgical scalpel.
 9. The surgical method of claim 1, wherein thehand-held surgical tool is a syringe with a surgical tool member that isa needle located at a distal end of the syringe.
 10. A surgical method,comprising capturing using an image capture device, image data thatincludes both a first detectable target on a hand-held surgical tool anda second detectable target proximate to a patient, wherein the firstdetectable target is located on the hand-held surgical tool at a firstpredefined target location, wherein the second detectable target islocated on the patient at a second predefined target location, andwherein a location relationship between a tissue of interest of thepatient and the second predefined location of the second detectabletarget is known; determining a location of the first detectable targetand an orientation of the first detectable target in a three dimensional(3D) spaced based on the image data; retrieving hand-held surgical toolmodel data that represents the hand-held surgical tool, wherein thehand-held surgical tool model data comprises first detectable targetlocation model data that corresponds to the predefined location of thefirst detectable target; determining a current location of the hand-heldsurgical tool and a current orientation of the hand-held surgical toolin the 3D space based on the determined location of the first detectabletarget and the determined orientation of the first detectable target;determining a location of the second detectable target in the 3D spacebased on the image data; and determining a location of the tissue ofinterest in the 3D space based on the determined location of the seconddetectable target on the patient, wherein the first detectable target ison a first surgical tool member of the hand-held surgical tool at thefirst predefined target location on the first surgical tool member ofthe hand-held surgical tool, wherein the hand-held surgical toolincludes a second surgical tool member that is coupled to the firstsurgical tool member, wherein a third detectable target is located onthe second member of the hand-held surgical tool at a third predefinedtarget location, and wherein the hand-held surgical tool model datafurther comprises third detectable target location model data thatcorresponds to the predefined location of the third detectable target,the method further comprising: determining a current location of thefirst member of the hand-held surgical tool and a current orientation ofthe first member of the hand-held surgical tool in the 3D space based onthe determined location of the first detectable target and thedetermined orientation of the first detectable target; determining alocation of the third detectable target and an orientation of the thirddetectable target in the 3D space based on the image data, and based onthe hand-held surgical tool model data; and determining a currentlocation of the second member of the hand-held surgical tool and acurrent orientation of the second member of the hand-held surgical toolin the 3D space based on the determined location of the third detectabletarget and the determined orientation of the third detectable target.11. The surgical method of claim 10, wherein a first surgical toolmember is disposed on a distal end of the first surgical tool member,wherein a second surgical tool member is disposed on a distal end of thesecond surgical tool member, wherein the first surgical tool member andthe second surgical tool member cooperatively operate in a coordinatedmanner to perform a surgical operation on the tissue of the patient, andwherein the method further comprises: determining a current location ofthe first surgical tool member in the 3D space based on the determinedlocation and orientation of the first surgical tool member, and based onthe hand-held surgical tool model data; and determining a currentlocation of the second surgical tool member in the 3D space based on thedetermined location and orientation of the second surgical tool member,and based on the hand-held surgical tool model data.
 12. A surgicalmethod, comprising: capturing using an image capture device, image datathat includes both a first detectable target on a hand-held surgicaltool and a second detectable target proximate to a patient, wherein thefirst detectable target is located on the hand-held surgical tool at afirst predefined target location, wherein the second detectable targetis located on the patient at a second predefined target location, andwherein a location relationship between a tissue of interest of thepatient and the second predefined location of the second detectabletarget is known; determining a location of the first detectable targetand an orientation of the first detectable target in a three dimensional(3D) spaced based on the image data; retrieving hand-held surgical toolmodel data that represents the hand-held surgical tool, wherein thehand-held surgical tool model data comprises first detectable targetlocation model data that corresponds to the predefined location of thefirst detectable target; determining a current location of the hand-heldsurgical tool and a current orientation of the hand-held surgical toolin the 3D space based on the determined location of the first detectabletarget and the determined orientation of the first detectable target;determining a location of the second detectable target in the 3D spacebased on the image data; and determining a location of the tissue ofinterest in the 3D space based on the determined location of the seconddetectable target on the patient, wherein the first detectable target ison a first surgical tool member of the hand-held surgical tool at thefirst predefined target location on the first surgical tool member ofthe hand-held surgical tool, wherein a third detectable target is on afirst surgical tool member of the hand-held surgical tool at a thirdpredefined target location on the first surgical tool member that isdifferent from the first predefined target location, wherein thehand-held surgical tool includes a second surgical tool member that iscoupled to the first surgical tool member, wherein a fourth detectabletarget is located on the second member of the hand-held surgical tool ata fourth predefined target location, wherein a fifth detectable targetis on a second surgical tool member of the hand-held surgical tool at afifth predefined target location on the first surgical tool member thatis different from the fourth predefined target location, wherein thehand-held surgical tool model data further comprises: third detectabletarget location model data that corresponds to the predefined locationof the third detectable target; fourth detectable target location modeldata that corresponds to the predefined location of the fourthdetectable target; and fifth detectable target location model data thatcorresponds to the predefined location of the fifth detectable target,the method further comprising: determining a location of the thirddetectable target in the 3D space based on the image data, and based onthe hand-held surgical tool model data; determining a location of thefourth detectable target in the 3D space based on the image data, andbased on the hand-held surgical tool model data; determining a locationof the fifth detectable target in the 3D space based on the image data,and based on the hand-held surgical tool model data; determining acurrent location of the first member of the hand-held surgical tool anda current orientation of the first member of the hand-held surgical toolin the 3D space based on the determined location of the first detectabletarget and the determined location of the third detectable target, andbased on the hand-held surgical tool model data; and determining acurrent location of the second member of the hand-held surgical tool anda current orientation of the second member of the hand-held surgicaltool in the 3D space based on the determined location of the fourthdetectable target and the determined location of the fifth detectabletarget, and based on the hand-held surgical tool model data.
 13. Asurgical method, comprising: capturing using an image capture device,image data that includes both a first detectable target on a hand-heldsurgical tool and a second detectable target proximate to a patient,wherein the first detectable target is located on the hand-held surgicaltool at a first predefined target location, wherein the seconddetectable target is located on the patient at a second predefinedtarget location, and wherein a location relationship between a tissue ofinterest of the patient and the second predefined location of the seconddetectable target is known; determining a location of the firstdetectable target and an orientation of the first detectable target in athree dimensional (3D) spaced based on the image data; retrievinghand-held surgical tool model data that represents the hand-heldsurgical tool, wherein the hand-held surgical tool model data comprisesfirst detectable target location model data that corresponds to thepredefined location of the first detectable target; determining acurrent location of the hand-held surgical tool and a currentorientation of the hand-held surgical tool in the 3D space based on thedetermined location of the first detectable target and the determinedorientation of the first detectable target; determining a location ofthe second detectable target in the 3D space based on the image data;and determining a location of the tissue of interest in the 3D spacebased on the determined location of the second detectable target on thepatient, wherein the first detectable target is on a first surgical toolmember of the hand-held surgical tool at the first predefined targetlocation on the first surgical tool member of the hand-held surgicaltool, wherein the hand-held surgical tool includes a second surgicaltool member that is coupled to the first surgical tool member, wherein athird detectable target is located on the second member of the hand-heldsurgical tool at a third predefined target location, wherein a fourthfirst detectable target is located where the first surgical tool memberis coupled to the second surgical tool member, wherein the hand-heldsurgical tool model data further comprises: third detectable targetlocation model data that corresponds to the predefined location of thethird detectable target; and fourth detectable target location modeldata that corresponds to the predefined location of the fourthdetectable target, the method further comprising: determining a locationof the third detectable target in the 3D space based on the image data,and based on the hand-held surgical tool model data; determining alocation of the fourth detectable target in the 3D space based on theimage data, and based on the hand-held surgical tool model data;determining a current location of the first member of the hand-heldsurgical tool and a current orientation of the first member of thehand-held surgical tool in the 3D space based on the determined locationof the first detectable target and the determined location of the fourthdetectable target, and based on the hand-held surgical tool model data;and determining a current location of the second member of the hand-heldsurgical tool and a current orientation of the second member of thehand-held surgical tool in the 3D space based on the determined locationof the third detectable target and the determined location of the fourthdetectable target, and based on the hand-held surgical tool model data.14. The surgical method of claim 13, wherein a first surgical toolmember is disposed on a distal end of the first surgical tool memberwherein a second surgical tool member is disposed on a distal end of thesecond surgical tool member, wherein the first surgical tool member andthe second surgical tool member cooperatively operate in a coordinatedmanner to perform a surgical operation on the tissue of the patient, andwherein the method further comprises: determining a current location ofthe first surgical tool member and a current orientation of the firstsurgical tool member in the 3D space based on determined location andorientation of the first surgical tool member, and based on thehand-held surgical tool model data; and determining a current locationof the second surgical tool member and a current orientation of thefirst surgical tool member in the 3D space based on determined locationand orientation of the second surgical tool member, and based on thehand-held surgical tool model data.
 15. The surgical method of claim 14,wherein the hand-held surgical tool is a surgical scissor, wherein thefirst surgical tool member is a first cutting edge of the surgicalscissor, and wherein the second surgical tool member is a second cuttingedge of the surgical scissor.
 16. The surgical method of claim 14,wherein the manually manipulated surgical instrument is a surgicalclamp, wherein the first surgical tool member is a first clamp of thesurgical clamp, and wherein the second surgical tool member is a secondclamp of the surgical clamp.
 17. A surgical method, comprising:capturing using an image capture device, image data that includes both afirst detectable target on a hand-held surgical tool and a seconddetectable target proximate to a patient, wherein the first detectabletarget is located on the hand-held surgical tool at a first predefinedtarget location, wherein the second detectable target is located on thepatient at a second predefined target location, and wherein a locationrelationship between a tissue of interest of the patient and the secondpredefined location of the second detectable target is known;determining a location of the first detectable target and an orientationof the first detectable target in a three dimensional (3D) spaced basedon the image data; retrieving hand-held surgical tool model data thatrepresents the hand-held surgical tool, wherein the hand-held surgicaltool model data comprises first detectable target location model datathat corresponds to the predefined location of the first detectabletarget; determining a current location of the hand-held surgical tooland a current orientation of the hand-held surgical tool in the 3D spacebased on the determined location of the first detectable target and thedetermined orientation of the first detectable target; determining alocation of the second detectable target in the 3D space based on theimage data; and determining a location of the tissue of interest in the3D space based on the determined location of the second detectabletarget on the patient, wherein the hand-held surgical tool is a syringewith a surgical tool member that is a needle located at a distal end ofthe syringe, wherein the first detectable target is on a plunger of thesyringe, wherein a second detectable target is on a barrel of thesyringe, and wherein the hand-held surgical tool model data furthercomprises: second detectable target location model data that correspondsto the predefined location of the second detectable target; and volumedata corresponding to a volume of the barrel of the syringe, the methodfurther comprising: determining a change in location of the firstdetectable target with respect to the location of the first detectabletarget; determining a change in a travel distance of a barrel seallocated at the distal end of the plunger, wherein the travel distanceequal a distance in the change of the location of the first detectabletarget relative to the second detectable target; and determining avolume defined within the barrel based on the determined travel distanceof the barrel seal.
 18. A surgical method, comprising: capturing usingan image capture device, image data that includes both a firstdetectable target on a hand-held surgical tool and a second detectabletarget proximate to a patient, wherein the first detectable target islocated on the hand-held surgical tool at a first predefined targetlocation, wherein the second detectable target is located on the patientat a second predefined target location, and wherein a locationrelationship between a tissue of interest of the patient and the secondpredefined location of the second detectable target is known;determining a location of the first detectable target and an orientationof the first detectable target in a three dimensional (3D) spaced basedon the image data; retrieving hand-held surgical tool model data thatrepresents the hand-held surgical tool, wherein the hand-held surgicaltool model data comprises first detectable target location model datathat corresponds to the predefined location of the first detectabletarget; determining a current location of the hand-held surgical tooland a current orientation of the hand-held surgical tool in the 3D spacebased on the determined location of the first detectable target and thedetermined orientation of the first detectable target; determining alocation of the second detectable target in the 3D space based on theimage data; and determining a location of the tissue of interest in the3D space based on the determined location of the second detectabletarget on the patient, wherein the surgical team member who is manuallymanipulating the manually manipulated surgical instrument is a firstsurgical team member that is viewing the rendered image on a firstdisplay, wherein a second surgical team member is controlling operationof a robotic surgery system that robotically manipulates a roboticsurgical instrument, wherein the rendered image includes the firstgraphical representation of the tissue of interest of the patient, thesecond graphical representation of the manually manipulated surgicalinstrument being manipulated by the first surgical team member, and athird graphical representation of the robotic surgical instrument,wherein the method further comprises: presenting the rendered image onthe first display that is being viewed by the first surgical teammember; and presenting the rendered image on a second display that isbeing viewed by the second surgical team member, and wherein therendered image is concurrently presented on the first display to thefirst surgical team member and on the second display to the secondsurgical team member.
 19. The surgical method of claim 18, furthercomprising: receiving robotic surgical instrument model information fromthe robotic surgery system; receiving robotic surgical instrumentlocation information from the robotic surgery system that is based on acurrent location of the robotic surgical instrument; and receivingrobotic surgical instrument orientation information from the roboticsurgery system that is based on a current orientation of the roboticsurgical instrument, wherein the rendered image, when presented on thefirst display and the second display, indicates the location and theorientation of the robotic surgical instrument relative to the locationand the orientation of the manually manipulated surgical instrument, andindicates the location and the orientation of the robotic surgicalinstrument relative to the organ of interest.